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SUMVIVRY 

In state needs assessment meetings held in late June 2008, the leadership of two Central Re- 
gion state departments of education, South Dakota and Missouri, asked REL Central to deter- 
mine how the states in the region evaluate their supplemental educational service (SES) pro- 
viders. Specifically, these SEA staff requested more information about Central Region states’ 
data collection processes, examples of instruments in use, and information on state SES con- 
tacts’ perceptions of their evaluation processes, including issues that have emerged and/or 
problems that have been solved. 

In response to this regional request, REL Central reviewed the federal guidance provided to 
states on monitoring SES providers and developed the following research questions: 

1. What criteria do states have for provider performance; how do they assess provider per- 
formance; and what policies do they have for terminating a provider? 

2. What systems do Central Region states use to gather information with which to eva- 
luate their providers? 

3. What do state contacts report about their experience with the state’s evaluation of pro- 
viders? 

States are required to develop standards for monitoring and evaluating providers. Only one 
state, South Dakota, has standards in place. The others rely on criteria that are found in the 
provider application, which defines the role and expectations of the provider. Typically, adhe- 
rence to the provider contract becomes a substitute for criteria in evaluating provider perfor- 
mance. South Dakota is the only state with a provider termination policy. 

The study found that the seven Central Region states include varying numbers of approved 
providers, eligible schools, and eligible students, which influences the scope and scale of their 
provider evaluations. State contacts were primarily concerned that their states did not have suf- 
ficient staffing to adequately monitor and evaluate the providers, but suggested specific aspects 
of evaluation and monitoring that they believed worked well in their states. 

Six of the states (all but Nebraska, which has an evaluation process in place but did not identify 
any schools qualified to receive SES in 2007) have made or are making changes to better eva- 
luate the performance and impact of their SES providers. 


mCNSROTJND 


Among the primary accountability measures of the No Child Left Behind Act of 2001 (NCLB) is 
the requirement for schools to make “adequate yearly progress” (AYP) toward bringing all of 
their students, categorized by subgroup, to proficiency in mathematics and reading by the end of 
the 2013-2014 school year. Schools that do not make AYP for two consecutive years are identi- 
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fied as being “in need of improvement,” or in School Improvement I status. Once identified as 
being in need of improvement, schools are required to offer low-income students the option of 
transferring to a school that has not been identified as being in need of improvement. Schools 
that fail to make AYP for three consecutive years are categorized as being in “School Improve- 
ment II” status. Title I 1 2 schools that fall into School Improvement II status are required to offer 
supplemental educational services (SES), in addition to the transfer option, to their low-income 
students (GAO August, 2006). 

These supplemental educational services, tutoring and other supplemental and academic enrich- 
ment, are provided free of charge to eligible students outside of the regular school day. Accord- 
ing to non-regulatory guidance issued by the U.S. Department of Education, such services must 
be “designed to increase the academic achievement of students in schools in need of improve- 
ment” (U.S. Department of Education, 2005, p.l). These services 

...may include academic assistance such as tutoring, remediation and other educational 
interventions, provided such approaches are consistent with the content and instruction 
used by the local education agency (LEA) and are aligned with the state’s academic con- 
tent standards. Supplemental educational services must be of high quality; research- 
based, and specifically designed to increase student academic achievement. (U.S. De- 
partment of Education, 2005. p.l). 

Supplemental educational services can be provided by a variety of entities, including non-profit 
groups, for-profit companies, local community programs, private schools, charter schools, na- 
tional organizations, faith-based groups, public schools and districts, and colleges or universities. 
However, providers must be approved by the state before they can begin offering services. 

A district with students who are eligible to participate in SES is responsible for informing par- 
ents of their children’s eligibility, of the providers available to them, and how to enroll their 
children in services. Some services are arranged and offered at the school after school hours; 
some are offered on Saturdays either at the school or at a provider site; some in the summer; and 
some are offered onhne, either at the school or through online courses taken at home. 

According to the Department of Education Supplemental Educational Services Non-Regulatory 
Guidance, 

SEAs have a responsibility, through the approval and monitoring processes, to ensure 
that high-quality services are delivered. In general, SEAs must identify the approved 
providers and determine whether providers improve student academic achieve- 
ment. ..Specifically, an SEA must develop and implement standards and techniques for 
monitoring the quality, performance, and effectiveness of the services offered by ap- 
proved supplemental educational service providers. Such standards and techniques, as 
well as any findings resulting from such monitoring, must be publicly reported. These 
quality control standards and techniques should be consistent with the initial criteria de- 


1 Title I schools are schools that operate programs funded under Title I of the elementary and Secondary Education 
Act of 1965 (ESEA). Title I, Part A includes the parental choice option of supplemental educational services. 

2 State Education Agencies 
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veloped for identifying potential providers [Section 1116(e) (4) (D)]. SEAs may also want to 
collect and report information about parent or student satisfaction with services (U.S. De- 
partment of Education, 2005, p. 18). 


A prior REL Central study (Barley & Wegner, 2007) identified eligible students’ participation in 
SES in Central Region states at 11 percent, below the national average of 17 percent. The 
present study describes how the seven states in the Central Region 3 4 are monitoring and evaluat- 
ing the providers of supplemental educational services in their states. These seven states, are cur- 
rently developing their evaluation systems, and like many other states (U.S. Department of Edu- 
cation, 2008) do not yet have a fully developed monitoring and evaluation program in place. The 
request for descriptions of what neighboring states were doing was intended to assist states in 
putting their systems in place. We expected the ‘work in progress’ nature of the evaluations to be 
reflected in this review. 

The non-regulatory guidance (U.S. Department of Education, 2005) suggests that parent and 
school support of the supplemental educational services program are elements that can potential- 
ly enhance student participation. In the guidance, SEAs are encouraged “to consider ways that 
[they] can help parents understand and access supplemental educational services for their child- 
ren.” (p.4); further, “SEA[s] should encourage LEAs 5 to provide teachers and principals in their 
district with information about supplemental educational services and local providers, so that 
these educators can act as a resource for parents” (p.5). The guidance also suggests that states 
may want to monitor parent or student satisfaction with services. 

Regional Importance 

In state needs assessment meetings held in late June 2008, McREL was asked by the leadership 
of two Central Region state departments of education, South Dakota and Missouri, to determine 
how the states in the region evaluate their SES providers: to describe their data collection and 
provide examples of instruments in use, and to describe the state SES contacts’ perceptions of 
their evaluation processes, including issues that have emerged and/or problems that have been 
solved. 


RESEARCH QUESTIONS 


In response to the regional request, McREL reviewed the federal guidance provided to states on 
monitoring SES providers and developed the following three research questions: 

1 . What criteria do states have for provider performance; how do they assess provider per- 
formance; and what policies do they have for terminating a provider? 


3 The Central Region includes the states of Colorado, Kansas, Missouri, Nebraska, North Dakota, South Dakota and 
Wyoming. 

4 The Central Region includes the states of Colorado, Kansas, Missouri, Nebraska, North Dakota, South Dakota and 
Wyoming. 

4 Local Educational Agencies 
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2. What system for gathering information do Central Region states use to evaluate provid- 
ers? 

3. What do state contacts report about their experience with the state’s evaluation of provid- 
ers? 

Data Collection 

To address the research questions, McREL collected data from SEAs on four monitoring re- 
quirements imposed by the U.S. Department of Education: 1) standards and techniques for moni- 
toring quality; 2) the system for gathering information; 3) how providers are to assess academic 
progress; and 4) how an SEA terminates approval of providers (U.S. Department of Education, 
2005; D-l through D-4, pp 18-19). Additionally, in response to the direct request from SEAs for 
the study, we asked SEAs about their experience with monitoring and evaluating the program. 
These data were used to prepare state-level data compilations which were used in creating the 
current report. 

Each SEA in the Central Region has designated a single state contact for their SES program. 

To address the original request for the study and obtain information on how each state in the 
Central Region monitors and evaluates SES providers, McREL conducted telephone interviews 
with each state agency SES contact person. State contacts were identified through the Center on 
Innovation & Improvement database (http://www.centerii.org/centerIIPublic/criteria.aspx) . All 
interviewees were informed that this cross-state report would use state names, but that they 
would have an opportunity to review their state’s information for accuracy. Only one state asked 
that specific data not be reported.. 

The interview protocol (Appendix A) was organized around the four federal monitoring require- 
ments and SEAs’ experiences in evaluating their programs. Each of the interviews was con- 
ducted by the same researcher. Given the emphasis on parent and school support, we asked about 
the provider communications with parents and schools, the provider’s efforts to work with them 
(provider collaboration ), and what data states collected on parent, school, and district satisfac- 
tion with the provider services. 


Data Processing and Analysis 

The interviewer recorded each state contact’s responses to the interview questions. McREL then 
compiled a list of the evaluation instruments the state uses and noted, based on the original re- 
quest, whether copies could be obtained and shared with the other states in the region. Next, both 
researchers reviewed and edited the recorded responses for completeness and accuracy for each 
state on a clean copy of the protocol. These documents became the seven individual state data 
compilations, which were organized in the same format as the interview protocol: the system for 
gathering data, data collected from the provider, other data used to evaluate/monitor providers, 
termination of approved provider, and overall perceptions of the state contact about the evalua- 
tion process used by the state. The primary purpose of the state-level data compilation was to 
confirm the information obtained in the phone interviews; however, in addition to a description 
of how the state evaluates providers, (the original impetus for this study), each includes a list of 
materials (i.e., evaluation instruments and sample letters) that the state has developed, along with 
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information on how to download copies of the materials from the state’s website. These state 
compilations were sent to each SEA contact to confirm the accuracy of the information pre- 
sented. Revisions were made if the state contact indicated that they were necessary. These seven 
state-level data compilations were then used by the researchers to synthesize findings across the 
seven states. 


FINDINGS?’ 

The seven Central Region states have varying numbers of approved providers, eligible schools, 
and eligible students. Table 1 displays 2007 data for the number of state-approved providers, the 
number of qualified Title I schools, and the number of eligible students in those schools. 


Table 1. Number of SES providers and eligible schools and students in the Central Region 
in 2007 



CO 

KS 

MO 

NE 

ND 

SD 

WY 

#Approved 

providers 

54 

17 

66 

5 

14 

17 

3 

# Qualified 
schools 

91 

35 

104 

0 

14 

30 

6 

# Eligible stu- 
dents 

35,435 

6,344 

32,181 

0 

554 

5,500 

1,116 


Note. Data obtained from phone interviews with state SES contacts 


To evaluate their providers, the seven states collect data from a variety of sources and on a range 
of topics. In some cases, states collect the data directly from the source (providers, parents, etc.); 
in others, states rely on reports from districts. Each SEA is required to monitor providers to de- 
termine whether they are increasing the academic achievement of students. The non-regulatory 
guidance suggests collecting provider data about parent and student satisfaction, student enroll- 
ment and attendance, frequency of progress reports to parents, and student achievement, either 
from their own assessments or from state achievement tests (U.S. Department of Education, 
2005). 

Table 2 summarizes findings from the first research question, “what criteria do states have for 
provider performance; how do they assess provider performance; and what policies do they have 
for terminating a provider?” In order to examine the second research question, “what system for 
gathering information do Central Region states use to evaluate providers?,” the researchers noted 
the nature of each state’s system and, for cross-state analysis, determined whether or not each 
data point collected by any one state was also collected by the other states (see Tables 3 and 4). 
To answer the third research question, “what insights and issues do state contacts have regarding 
the state’s evaluation of providers?” researchers reviewed state contacts’ responses to six ques- 
tions about their experience with SES provider monitoring and evaluation for similarities and 
differences. 


6 All of the information in the Findings section is derived from the state contact interview data. 
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What criteria do states have for provider performance, how do they assess 
provider performance, and do they have a policy for terminating a provid- 
er? 


In 2008, the U.S. Department of Education produced an interim report on state implementation 
of NCLB, which included Volume IV on SES. That report states that 

As of 2004-2005, most states were working to develop and implement standards for the 
monitoring and evaluation of supplemental service providers; only a small number of 
states had statewide databases incorporating participation and achievement information 
that would permit rigorous evaluations of providers’ effects, (p.xvi) 

Although all seven states in the Central Region collect data to evaluate their providers, the crite- 
ria used to interpret the data are not always clear, nor are they consistent across states. All state 
contacts were asked about provider performance standards and their policies for terminating pro- 
viders. Their responses are summarized in Table 2. Of the seven Central Region states, only 
South Dakota reported that they had established standards for providers and potential providers 
and posted them on their website. 

In using the data from district reports or data collected directly by states, states assess provider 
performance in a variety of ways. For example, North Dakota has developed a Self Monitoring 
Guide for Providers that must be completed annually and is then reviewed by the SEA. If the 
Department of Public Instruction determines that it is necessary, SEA staff will make a site visit 
to the provider. South Dakota has hired a monitor to make loosely structured site visits to 
schools to monitor supplemental services programs. In the absence of performance standards, the 
remaining five states rely on assurances in the provider application and/or the contract to estab- 
lish their expectations of the providers (criteria for performance). Reviewing provider com- 
pliance with these criteria varies; two states, Kansas and Wyoming, require providers to reapply 
every two years, and South Dakota holds a mandatory summer meeting of providers with dis- 
tricts to ensure that both understand and adhere to the program requirements. 

Only one of the states in the Central Region, South Dakota, has a policy for terminating provid- 
ers, although another state is working with their attorney to develop such a policy. No state has 
terminated a provider, and to date, in only one instance did a state permit a dissatisfied school to 
discontinue their relationship with a provider. 


Table 2. State policies for monitoring SES providers 



CO 

KS 

MO 

NE 

ND 

SD 

WY 

Performance 

Standards 

N 

N 

N 

N 

N 

Y 

N 

Termination 

Policy 

N 

N 

N 

N 

N 

Y 

N 


Note. Data obtained from phone interviews with state SES contacts 
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What system for gathering information do Central Region states use to eva- 
luate providers? 


Central Region states vary considerably in how they collect and use data to evaluate providers 
(see Tables 3 and 4). A detailed listing by state of instruments used, data sources, administering 
agencies, and the frequency with which data are used, is included in Appendix B. 

The variables states collect vary. Provider communication includes letters, emails, telephone 
calls, and face-to-face encounters with both parents and schools. The frequency and content of 
communications are thought to influence student participation (U.S. Department of Education, 
2008). Provider collaboration, working with parents and schools to determine aspects of the 
program, also has the potential to influence parent satisfaction. Perceptions of satisfaction with 
provider services and how often a provider reports students’ progress are examples of data that 
the non-regulatory guidance suggests states could collect to assess provider effectiveness (U.S. 
Department of Education, 2005). For some states, these variables serve in the evaluation as lead- 
ing indicators toward the intended outcome of improved student academic achievement. States 

operationalize these variables by asking a variety of questions, such as Are you happy with 

Or are you satisfied with ?, or simply by asking respondents to rate the variable on a five- 

point scale. 

States in the Central Region, directly or through districts, collect data from parents, school prin- 
cipals, and site coordinators. They also collect student data from providers. As illustrated in 
Tables 3 and 4, not every state collects data from each group. For example, to date Colorado has 
relied entirely on student data to evaluate providers’ performance. Missouri and Wyoming, on 
the other hand, have not included student results in their evaluations. Instead, Missouri has eva- 
luated the effectiveness of the providers’ communications and collaboration through a review of 
materials solicited from the provider. These materials include surveys of providers, parents, 
teachers, principals, and district coordinators. Missouri’s Provider Questionnaire asks providers 
to describe the program, including the tutoring sessions and attendance, as well as the provider’s 
communications with parents and schools. Missouri also asks providers to evaluate parent and 
school participation and to evaluate their own performance. The state then hires an outside eva- 
luator to review the materials and prepare a report based on these data. Wyoming has distributed 
surveys to schools for parents and teachers to complete, but has had little success in getting res- 
ponses; only one of the six Wyoming schools contacted in 2007 responded to the survey. 

Table 3. Data sources and variables collected by Central Region states directly or from dis- 


tricts to evaluate SES provicl 

lers 


CO 

KS 

MO 

NE 7 

ND 

SD 

WY 

Data Collected from Parents 

Provider Collaboration 

No parent 

No par- 

Y 

Y 

Y 

Y 

Y 


7 Nebraska has had only one school eligible for SES and it has not yet provided services. Therefore, their system has 
not yet been used. 
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CO 

KS 

MO 

NE 7 

ND 

SD 

WY 

Provider Communication 

survey 

ent sur- 
vey 

Y 

Y 

Y 

Y 

Y 

Provider Effectiveness 

Y 

Y 

Y 

Y 

Y 

Data Collected from Parents 

Satisfaction with provider 



Y 

Y 

Y 

Y 

Y 

Data Collected from School Principal or Site Coordinator (for Wyoming - Teacher) 

Provider Collaboration 

No 

school 
level sur- 
vey 

Y 

Y 

No 

school 

level 

survey 

Y 

N 

Y 

Provider Communication 

Y 

Y 

Y 

N 

Y 

Satisfaction with provider 

Y 

Y 

Y 

N 

Y 

Data Collected from Districts 

Provider Collaboration 

No dis- 
trict sur- 
vey 

No dis- 

trict 

survey 

Y 

Y 

Y 

N 

Y 

Provider Communication 

Y 

Y 

Y 

N 

Y 

Provider Effectiveness 

Y 

Y 

Y 

Y 

Y 

Satisfaction with provider 

Y 

Y 

Y 

Y 

Y 


Note. Data obtained from phone interviews with state SES contacts 


Under NCLB, SEAs must “determine whether the services that providers offer to students are 
contributing to increasing the academic achievement of students.” (U.S. Department of Educa- 
tion, 2005, p. 18) As shown in Table 4, there is considerable variation in the SES-related student 
data collected from providers by states in the Central Region. Providers may administer pre- and 
post-tests to measure student achievement; report attendance in tutoring sessions; indicate the 
number of students who have completed the program (based on the provider definition); and/or 
indicate the number of students that have achieved individual goals set within the program. 

North and South Dakota collect data on all four of these categories; Colorado requires providers 
to enter data online on pre-post test results and attendance. Missouri and Wyoming, on the other 
hand, have not included student data in their evaluations in 2007. Kansas and Nebraska (in its 
evaluation plan) only collect data pertaining to students’ achievement of their goals. 

As states move toward establishing systems that assign unique student identification numbers, it 
may be possible to more easily track the impact of SES providers on student achievement. Once 
student identification numbers are in place, for example, states and providers can gauge the im- 
pact of their services, for the students who receive services, by provider organization and by the 
nature and amount of service. Assessment data for participating students can be compared to as- 
sessment data for students who do not participate in SES, to rule out increases from growth in 
achievement over time rather than increases associated with provider services. Currently, the 
student data available are from pre- and post-tests administered by providers. These vary from 
provider to provider, and are not necessarily aligned with either classroom instruction or state 
standards. 

Table 4. Student Data collected by provider and reported for individual students 


CO 

KS 

MO 

NE 

ND 

SD 

WY 
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Pre/post tests 

Y 

N 

N 

N 

Y 

Y 

N 

Attendance 

Y 

N 

N 

N 

Y 

Y 

N 


CO 

KS 

MO 

NE 

ND 

SD 

WY 

Completion of 
Program 

N 

N 

N 

N 

Y 

Y 

N 

Achievement of 
Goals 

N 

Y* 

N 

Y 

Y 

Y* 

N 


Note. Data obtained from phone interviews with state SES contacts *Group percents only are collected. 


The nature and size of states’ evaluations are determined by the number of eligible schools and 
providers that states are required to monitor (See Table 1), but also by the state’s ability and/or 
willingness to expend non-SES related resources toward the evaluation. Colorado and Missouri, 
which have more than 50 approved providers and about 100 schools participating in the SES 
program, have used an outside evaluator to assist with the program evaluation. In 2007, Colorado 
hired an outside evaluator to analyze student achievement data from the state test to evaluate 
providers. As previously noted, Missouri hired an outside evaluator to review their evaluation 
data. The South Dakota Department of Education, with fewer providers and schools, has hired a 
monitor to make on-site visits, and plans to hire a second monitor, in order to visit all 30 of their 
implementing schools. North Dakota, with 14 schools and 14 providers, administers a compre- 
hensive evaluation system in which they collect all of the data and documents used in the deli- 
very of services, which they then use to assess each provider’s performance. 


What do state contacts report about their experience with the state’s evalu- 
ation of providers? 

This research question addresses the regional interest in what issues other states are encountering 
and how they are addressing them. In order to provide some structure to state contacts’ responses 
to this particular research question, we asked six specific questions, as follows: 

In your perception of your state’s SES program, 

1. Are students gaining?; 

2. What has worked well?; 

3. What has not worked well?; 

4. What should the state have done differently?; 

5. What would you wish the state to have done? 

6. What would you not do again? 

In addition, we asked a final, open-ended question: “Is there anything else I should know?” 

Because Nebraska has not yet implemented their evaluation, their state contact was not able to 
respond to these questions. 

No state had a data report in 2007 to indicate whether or not students made achievement gains by 
participating in the SES program. Two of the six state contacts for this question indicated that it 
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was difficult to know whether or not students were making gains. Of the remaining four contacts, 
the South Dakota contact relayed anecdotal reports from schools and parents that indicated that 
students were making gains. The Wyoming contact indicated that students were not completing 
the program, which in the opinion of that contact would make progress unlikely. The North Da- 
kota contact believed that students should make gains if they participate in services; and the 
Kansas contact indicated that the extra assistance with academics should help students increase 
their achievement. This contact also felt that a locally based provider from within the district 
was more successful than a provider from out of the district’s area. 

Regarding what worked well, Kansas and Missouri both reported that it is too early to say. The 
remaining four states shared specific examples of processes or actions that had worked well in 
conducting their evaluations: attaching the reporting requirement to a regularly received report; 
requiring providers to reapply every two years; using an online data entry system to improve data 
retrieval; holding an SES conference for districts and providers annually and allowing the two 
groups to meet; using a monitor to conduct on-site visits; and using resources such as PIRC 
(Parent Information Research Center) to help develop parent materials. 

When asked what did not go well, three of the six contacts noted a lack of adequate staffing and 
resources at the state level. Two contacts indicated that there was not enough staff time allocated 
to monitor the program effectively; the third indicated that staffing was shared such that five dif- 
ferent persons worked on the program, with no consistency from one to the next. The North Da- 
kota contact mentioned the difficulty in starting from scratch to develop the evaluation aspect of 
the program, and the South Dakota contact expressed disappointment that districts had not sub- 
mitted parent materials for review, which in the contact’s view would have made them more ef- 
fective. The Colorado state contact pointed out that the lack of state assessment data for students 
in the lower grades meant that were no data available for 50% of the participating students. 

Things to do differently in the future include getting key materials out earlier; developing poli- 
cies to guide decision making; better on-site monitoring at the school level; and having resources 
to allocate more staff time to the program. Other suggestions included holding a required provid- 
er meeting to clarify program rules and to create an online reporting system. 

No state contact reported full confidence in the evaluation data and process currently in place, 
but all reported that they are considering ways to improve their evaluation processes. Three 
states reported expanding the use of an outside evaluator to obtain more reliable evaluation data. 
No state has yet developed a report on the supplemental services program that includes the in- 
formation needed to make decisions about how to improve the program. Nonetheless, most are 
seeking ways to improve how they evaluate their providers, and all have been willing to make 
their instruments available to other states. 

LEVITATIONS 


A limitation of this study is that interviews were conducted with a single contact person in each 
of the seven states. The report does not include the opinions of other state-level staff, district ad- 
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ministrators, providers, school personnel, parents or students. Further, given that none of the 
state contacts are devoted full-time to the program, the contacts surveyed for this study may have 
been limited in their ability to know all dimensions of the program. Although every effort has 
been made to cross-check the information received from the state contacts, this report is primari- 
ly a descriptive study of the opinions and perceptions of state SES contacts. 


DISCISSION 


The seven states in the Central Region vary in the scale and scope of their SES programs. The 
number of approved providers and eligible schools ranges from a low of three providers and six 

o 

qualified schools in Wyoming to a high of 66 providers and 104 qualified schools in Missouri. 
This influences the scope of the evaluations that they need and are able to conduct. 

States are required to develop standards for monitoring and evaluating providers. Only one state, 
South Dakota, has standards in place. The others rely on criteria that are found in the provider 
application, which defines the role and expectations of the provider. Typically, adherence to the 
provider contract becomes a substitute for criteria in evaluating provider performance. South 
Dakota is the only state with a provider termination policy. 

There are commonalities across the region in the systems states use for gathering evaluation data 
about SES providers. Basing their evaluations on the non-regulatory guidance has generally re- 
sulted in states collecting variables that are similar in name, although they often use different 
sources for the data. Providers are required to report student progress to the state; three states, 
Colorado and North and South Dakota received progress data from providers on provider admi- 
nistered tests as well as student attendance at services. Kansas only received data on student 
achievement of goals, and two states, Missouri and Wyoming, had not yet received student data 
from providers in 2007. 

State contacts were primarily concerned that their states did not have sufficient staffing to ade- 
quately monitor and evaluate the providers, but suggested specific aspects of evaluation and 
monitoring that they believed worked well in their states. 

Six of the states (all but Nebraska, which has an evaluation process in place but did not identify 
any schools qualified to receive SES in 2007) have made or are making changes to better eva- 
luate the performance and impact of their SES providers. 


8 There are five providers, but no eligible schools, in Nebraska. 
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Behind Act, Volume IV - Title I School Choice and Supplemental Educational Services: 
Interim Report, Washington, D.C. 
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APPENDIX A: P’ROTOCOL FOR STATE CXJNTACT INTERVIEWS 


Hello. My name is and I am calling on behalf on McREL. Two neighboring states 

have asked if we could leam from each of the seven states in the region how they are handling 
the evaluation of SES providers required in the guidance related to Supplemental Educational 
Services. Would you be willing to give me 20 to 30 minutes of your time to answer questions 
about your state’s evaluation process? I will share the results of these interviews with you and 
any examples or useful tips I leam in the process so that you may verify that our discussions 
were accurate. Due to an imposed deadline for the completion of this report, I will get the inter- 
view notes back to you within one day of the interview so that you can have two days to review 
and get the get back to me with your approval of the notes which I took during our conversation. 

The questions are in the four areas of the federal guidance that relate to Monitoring providers: 1) 
the system for gathering information, 2) standards and techniques for monitoring quality, per- 
formance, and effectiveness of the services, 3) how providers are required to demonstrate student 
progress, and 4) how approval of a provider could be terminated. I am also interested in your 
experience with evaluation/monitoring and what you have learned as a state contact. 

I reviewed the national database maintained at the Center for Innovation and Improvement for 
information you have submitted in the past about Provider Evaluation. I have retrieved the mate- 
rials from those links so we can refer to them as needed. 

They are listed below: 

Materials on State Monitoring and Evaluation of SES from the Center on Innovation & 
Improvement Database: State’s Response Retrieved July 20, 2008 from: 
http://www.centerii.org/centerIIPublic/criteria.aspx 


I. System for gathering information: 

• First, I would like to know if your state has established a set of standards which were developed 
for monitoring and evaluating the SES provider. Are you willing to share this document? 

• Please briefly describe how your state monitors/evaluates the providers. Probe for all kinds of 
data collection - from whom about what. Ask for copies of instruments if willing. (McREL will 
not share them if state prefers - but we need them to categorize variables covered and sources) 

• If an outside consultant is utilized to gather data, please ask permission to discuss this survey 
with them if needed. 


Name of Outside Consultant- Phone Number: 

Contact Person 
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Data Col- 
lection 
Event: 

Purpose: 

Source of 
data 

Who col- 
lects: 

Tim- 

ing/frequen 

cy 

Instrumenta- 

tion/Method 

How is it used in 
reports 

Example: 
Parent 
satisfac- 
tion sur- 
vey 

Obtain parent 
evaluation of 
provider 

Parents of 
enrolled stu- 
dents 

School 

Annually 

12 item survey using 
5 point likert scale 
from strongly disag- 
ree to strongly agree 

Sole source of data for 
parent evaluation sec- 
tion. 

# 1 







#2 

Etc. 








Note: When the purpose (for a particular data collection event) is to evaluate stu- 
dent progress and the provider is the collector, go to in, otherwise continue with 

n. 

Let’s use the Data Collection Events from above and begin to clarify the specific 
types of data that are obtained in each collection event. (Review and get clarifica- 
tion on all of the data they have to use in evaluating and ask for the topics covered 
in those data Underneath each data collection column, probe for types of data that 
are obtained and in what format ( i.e. how are respondents selected ( entire popu- 
lation or sample), perception data based on likert scales ( etc)). 


Assessing the variables 
in each data collection 
event? 

Data Collec- 
tion #1 

Data Collec- 
tion #2 

Data Collec- 
tion #3 

Data Collec- 
tion #4 

Data Col- 
lection #5_ 





Student Achievement 






Student Achievement 
pre/post as a group ( 
grade level) 






Achievement pre/post 
for individual students 






Student Attendance 






% students who achieved 
goals 






% students who made 
progress 






% students who showed 
no improvement 






No. who complete 
program 
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Special Ed/LEP Stu- 
dents 






# SpEd students served 






#LEP /ELL students 
served 






Provides services to 
LEP/ELL students 






Provides services to 
Spec Ed students 






Local and State Stan- 
dards 






Ensures the curriculum 
and instruction was con- 
sistent with classroom 
instructional program 






Ensures the curriculum 
and instruction was con- 
sistent with district’ s 
instructional program 






Ensures the curriculum 
and instruction was con- 
sistent with state stan- 
dards 






Quality of Instruc- 
tion/Instructors Pro- 
vided 






Provider works collabo- 
ratively with school and 
parent 






Overall Quality of In- 
struction in the Program 






Overall Quality of In- 
structors utilized by Pro- 
viders 






Ensures safety measures 
were built into the pro- 
gram 






Communication 






Provider utilizes Effec- 
tive Communication 
skills 






Provider communicates 
frequently regarding stu- 
dent achievement 






Provider Overall 






Utilizes research based 
programming 
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Provider Overall 






Overall Effectiveness of 
Program 






Provider Fulfills Con- 
tractual obligations 






Overall Satisfaction with 
Program 







II. Standards and techniques for monitoring quality, performance, and effectiveness of the 
services. 



How are 

respondents 

selected? 

What methods 
are utilized to 
obtain res- 
ponses from 
this group 

What is 

your 

average 

response 

rate from 

this 

group 

How are 
the data 
recorded? 

How are 
the data 
filed? 

Do you 

combine 

data from 

another 

source? 

Which? 

How then do 
you weight 
these data rela- 
tive to other 
sources? 

Sample: 

Parent 

Satisfaction 

Survey 

All parents 
of enrolled 
students 

Survey mailed 
to homes, fol- 
low up tele- 
phone call 

75% 

Summed 
by hand 

Summary 
kept in 
provider 
file 

NA 

NA 

#1 








#2 








#3 









III. How providers are required to demonstrate student progress: 

• Ask these questions directly 


Specific Questions 

Specific Answers 

General Comments 

Do you require provid- 
ers to report to you or 
to the district? 



What do you require 
Providers to demon- 
strate? 



*What testing instru- 
ments do they use or 
are they required to 
use? 



What are the sche- 
dules for data collec- 
tion from the provider? 



Do you require them to 
only report a sample of 
the population or the 
entire population? 



What are your re- 
sponse rates like from 
the providers? 
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Specific Questions 

Specific Answers 

General Comments 

Do you make a report 
to the districts, etc 
regarding the evalua- 
tion of the providers? 




*Examples: Provider-made tests, nationally normed tests, school developed tests, state- 
developed tests 


IV. Termination of Approved Provider. 


Question 

Answer 

Have you terminated a provider for non- 
performance? 


On what basis did you determine termination? Are 
there policies in place for such an action? Could 1 
obtain a copy of them? 


If you have not terminated a provider yet, how would 
you make such a determination? Are there policies 
in place for such an action? Could 1 obtain a copy of 
them? 



V. I am also interested in your overall experience with evaluation/monitoring and what you 
have learned as a state contact. 


Open ended. 


Potential Probes: 

Answers 

1. Do you feel that Students have gained in know- 
ledge after exposure to the SES tutoring pro- 
gram? 


2. What can you report that has worked very well 
in the process of monitoring and evaluation of 
SES providers? 


3. What can you report that has not worked well 
in the process of monitoring and evaluation of 
SES Providers? 


4. Things that you feel the state should have done 
with the SES tutoring program. 


5. Things you wish the state would have done with 
the SES tutoring program. 


6. Things that you don’t think you will do again 
related to the SES tutoring program. 


7. Anything else that you would like to discuss that 
1 have not asked? 



VI. Conclusion Thank you for the interview today. I will be sending you a copy of my notes ( by email) for 
you to confirm our conversation in the next 48 hours. I would like to request that you return the document with your 
approval within 48 hours of receiving the notes. I apologize for the short turnaround, but it is necessary to meet the 
deadlines for this report, which will be made available to all states in the McREL region: Colorado, Kansas, Mis- 
souri, Nebraska, North Dakota, South Dakota and Wyoming. 


Copies of documents that the state has agreed to share 
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Name of Document 

Willing to share 
with States and 
McREL 

Willing to share with 
McREL only 

Available on 
the Website 

State contact will email 
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APPENDIX B 


Summary of Data Collected in Central Region States during the 2007-2008 School Year 



Colorado 

Missouri 

North Dakota 

South Dakota 

Wyoming 

Kansas 


Who Collects? 

Frequency 

Who Collects? 

Frequency 

Who Collects? 

Frequency 

Who Collects? 

Frequency 

Who Collects? 

Frequency 

Who Collects? 

Frequency 

District 

Survey 

Not Collected 

NA 

Consultant sends to 
District Coordinators — 
completed online 

A 

State collects in- 
formation from 
district/school on 
Annual Program 
Improvement Re- 
port 

A 

State gives to the 
district to admi- 
nister. District is 
returns to the 
state. 

A 

State gives to the 
district to administer. 
District is returns to 
the state. Note: No 
district had respond- 
ed to this in 07-08 

A 

State sends 
surveys to 
district 

A 

School 

Survey 

Not Collected 

NA 

Consultant sends to 
School principals or site 
coordinators — survey 
completed online 

A 

State collects in- 
formation from 
district/school on 
Annual Program 
Improvement Re- 
port 

A 

State gives to the 
district to admi- 
nister. District is 
returns to the 
state 

A 

Not Collected 

NA 

Not Col- 
lected 

NA 

Class- 

room 

Teacher 

Survey 

Not Collected 

NA 

Consultant sends to 
classroom teacher and 
they complete the sur- 
vey online for each 
provider 

A 

Not Collected 

NA 

State gives to the 
district to admi- 
nister. District is 
returns to the 
state. 

A 

State gives to the 
district to administer. 
District is returns to 
the state. 

A 

Not Col- 
lected 

NA 

Parent 

Survey 

Not Collected 

NA 

Consultant sends paper 
copies to the District 
Coordinator who then 
gives them to school 
principals who are to 
distribute them to the 
students to take home. 
School collects them 
and sends back via US 
mail. 

A 

State sends forms to 
district-district col- 
lects and sends back 
to the state 

A 

State gives to the 
district to admi- 
nister. District is 
returns to the 
state.. 

A 

State gives to the 
district to administer. 
District is returns to 
the state. 

A 

Not Col- 
lected 

NA 
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Colorado 

Missouri 

North Dakota 

South Dakota 

Wyoming 

Kansas 

Provider 

Survey 

Provider in- 
puts data on a 
state devel- 
oped online 
database 

M 

Consultant sends to 
Providers who complete 
the survey online 

A 

State sends Provid- 
er Annual Report to 
district-district col- 
lects and sends back 
to state 

State sends Self - 
Monitoring Guide 
directly to Provider 
and collects annual- 
ly 

A 

State gives to the 
district to admi- 
nister. District is 
returns to the 
state. The state 
also has a survey 
that is collected 
from the tutors 
for the Providers 

A 

State Plans to collect 
information directly 
from Provider — had 
not completed this 
task as of 07-08 

A 

State sends 
surveys to 
Provider 

A 

State 

Coordi- 

nator 

Not Collected 

NA 

Consultant sends State 
Coordinator the survey 
to be completed online 

A 

Not Collected 

NA 

Not Collected 

NA 

Not Collected 

NA 

Not Col- 
lected 

NA 

Pre/Post 
Test Re- 
sults 

Provider 

M 

Not Collected 

NA 

Provider 

A 

State gives to 
district-district 
collects from 
Provider and then 
districts return to 
state 

A 

State Plans to collect 
information directly 
from Provider — had 
not completed this 
task as of 07-08 

A 

Not Col- 
lected 

NA 

Student 

Atten- 

dance 

Provider 

M 

Not Collected 

NA 

Provider 

A 

State gives to 
district-district 
collects from 
Provider and then 
districts return to 
state 

A 

State Plans to collect 
information directly 
from Provider — had 
not completed this 
task as of 07-08 

A 

Not Col- 
lected 

NA 

% / No. 
of stu- 
dents who 
achieve 
goals 

Provider 

M 

Not Collected 

NA 

Provider 

A 

State gives to 
district-district 
collects from 
Provider and then 
districts return to 
state 

A 

State Plans to collect 
information directly 
from Provider — had 
not completed this 
task as of 07-08 

A 

Provider 

A 

Who 

Analyzes 

Data? 

Outside Consultant: 
OMNI Evaluation in 
Denver CO 

CREP ( Center for Research 
and Educational Policy) 
University of Memphis 

State 

Institute for Educational 
leadership and Evalua- 
tion in Rapid City, SD. 

State 

State 


Note: Frequency: M = Monthly, A=Annually. Source: Phone Interviews with State Contacts during the fall of 2008-state contacts based all answers on the 2007-2008 school year 
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